Introduction
I get a lot of questions each month about user acceptance testing (UAT) - what it is, how to perform it, which (if any) tools to use, and a variety of other questions. The purpose of this article is to build on a base of past articles and continue to build a knowledge base and lessons learned. Who knows? One day all of these articles may turn into a book!
The keys presented in this article are not hidden from view, just often overlooked by people. If you ignore them, you will experience problems and less than effective user acceptance testing. If you heed them, you will open new doors to involving your users in testing.
User Acceptance Testing (UAT) Defined - User acceptance testing is the validation that a system or application will meet user needs in the operational or business environment.
This article can be applied from multiple perspectives:
- If you are a tester or test facilitator, this article is written directly to you. Your job is to help users plan and conduct an effective test to validate that the application will work correctly in their environment.
- If you are a user, focus the points in this article to you and other users that will be involved in the test.
- If you are a project manager, this article can provide valuable ways to involve users in the entire project, not just testing.
- If you are a developer, this article can help you understand what the users will need to do in testing the application. You can use the secrets to guide users in planning their test.
Key #1 - Understand that UAT is Performed at the Worst Possible Time in the Project
Most UAT efforts happen at the end of the project because this is when the entire system is assembled or installed. Until the end of the project, users may be able to test parts of the system or application, but not the system as a whole. This is bad because the end of the project is the worst time to find and fix major problems. Each problem found and fixed in system testing or UAT which has been in the system since requirements has a 10x cost factor had it been found or fixed in requirements or design. This is due to the ripple effect that the fix may require in other areas of the system.
The solution is to involve users throughout the project from the very beginning. When users are providing input to user requirements, they can also be defining acceptance criteria and can be involved in requirement reviews and inspections.
Key #2 - Base the Test on Real-world Conditions, Not User Requirements
This is one of the most controversial things I teach about UAT, but if you don't base the test on real-world conditions you are missing the point of UAT. There are two sides to testing - verification and validation. Verification is testing based on specifications and requirements. Validation is testing based on real-world operational conditions. You need to test from both perspectives. Validation is necessary because requirements have defects. In many cases, the requirements are not available, such as in the case of vendor-developed software.
The solution is to have users design tests that model their world. The test is to determine if the system or application will correctly support the real world conditions.
Key #3 - Understand Your Users
UAT is not a "one-size fits all" activity. Some user groups will not have the motivation, time or skills to design and perform an adequate test. Some user groups will be able to perform a very extensive test.
One solution is to profile the affected user groups. I often categorize users as:
- Not motivated or skilled - These people may not be against the UAT effort, but they just may not be aware of how to perform UAT or understand the importance of UAT. These users may lack basic computer literacy or management support to allow them to participate in the test. This is most often the level where surrogate users may be needed for testing. In addition, the tests may be minimal at this level.
- Somewhat motivated, but lacking skills - At this level you have a chance to get involvement but will have to build testing skills. Tests may range from low to moderately complex.
- Somewhat motivated and skilled - At this level the skills are in place, but you will have to identify motivating factors and market those to the prospective testers. Tests may range from low to high complexity.
- Very motivated, but lacking skills - The users in this group are ripe to learn to new things and contribute to the project. That's a good combination! Good training will often complete this picture. At this level, these people may perform tests in the minimal to moderately complex range.
- Very motivated and skilled - These are like "instant testers." Your training efforts will likely be minimal with this group and you shouldn't have to spend a lot of time motivating them. The challenge is to use their talents wisely and not burn them out during the test. These people can perform a full range of tests from low to high complexity.
Key #4 - Involve Your Users
Some organizations perform UAT with surrogate users - people who take the role of users but who are not the actual people in the field that will eventually use the application. The risk with this is that when the system is deployed, the real users will find problems the surrogates didn't consider. I only recommend this approach when the actual users are unavailable or unwilling to participate in the test. Even then, the risks are high.
The solution I advise is to at least hold limited review sessions with the actual users. Complete review sessions which examine items such as user requirements in detail are even better. Also, have contingency plans in place when unexpected problems are found during UAT. If users are unwilling or unable to participate in the project, raise this situation as a risk in the project status reports.
Key #5 - Match the Intensity of the Test to the Relative Risk and the Skills of the Users
Not every project requires extensive testing. However, for those projects that control high levels of assets or affect personal safely, extensive validation is required. Users and others on the project may question the need for defined test cases and test scripts, but when viewed in the light of project and business (or operational) risks, the time and resources spent in effective testing are resources well spent.
To match testing to the risk, perform a risk assessment that can be quantified and documented. Just guessing at the level of risk is not good enough to explain after a critical failure why you thought something was a low risk of failure. The risk assessment should indicate which system and business areas are the most exposed to risk. This allows test resources to be allocated where they will have the greatest impact to detect defects that may have severe negative consequences.
Key #6 - Plan the Test in Advance
There are three levels of test planning:
- The strategic level, which specifies what is to be performed in testing. The test strategy describes high-level direction and objectives, but stops short of "how" the test will be performed.
- The tactical or logistical level, which is also considered high-level, but describes how the test will be performed. This is basically the project plan for the test and is often written to focus on one phase of testing, such as unit, integration, system, or UAT.
- The detailed test case or test script level, which defines in detail the actions to be performed, the expected results and the procedures to perform the tests.
Some people prefer to use informal and random approaches to testing, which can find obvious defects but will often miss the more deeply embedded defects that are often the most troublesome. Another problem with a lack of test planning is that you never quite know for sure when you have completed the test. A test plan contains measurable objectives and pass/fail criteria. Detailed test plans also make it possible for one person to design the test and for many people to perform it. Finally, detailed test plans give you a way to recreate a test if you have to perform it again.
Once again, the extent of your test planning effort should be reasonable in light of the relative risks.
Key #7 - Review Your Test Plans
Test plans can have errors just like any other type of project documentation. UAT plans can be reviewed by the UAT team, a Quality Assurance (QA) team or facilitator, the project manager, or any other people with knowledge of testing and the project.
Conclusion
User acceptance testing is not trivial or easy. UAT can be one of the most critical and risky types of test on a project, which means that a great deal of care should be taken when planning, executing and evaluating the results of UAT. These keys of UAT have worked for other organizations in planning and performing UAT and they can work for yours as well.
